Skip to content

feat: per-user secret upstream headers for MCP registries (ENG-3243) - #1351

Open
edukmx wants to merge 16 commits into
developfrom
feat/eng-3243-per-user-secret-headers
Open

edukmx wants to merge 16 commits into
developfrom
feat/eng-3243-per-user-secret-headers

Conversation

@edukmx

@edukmx edukmx commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Refs ENG-3243 — https://linear.app/neuraltrust/issue/ENG-3243

Summary

Many upstream MCP servers do not speak OAuth and authenticate each person by their own API key in a custom header. Until now the gateway could only send static headers, the same for every caller. A registry can now declare per-user upstream headers: each person enrolls their own value once, it is stored encrypted in the vault, and the gateway sends the calling person's value on every request to that upstream. The key never leaves the gateway.

Declaration — mcp_target.user_headers (custom and catalog registries): name, description, required, secret, scheme (e.g. Bearer). Validation rejects reserved names (Host, Content-*, hop-by-hop, Cookie, Mcp-Session-Id, Mcp-Protocol-Version, X-NeuralTrust-*, …), duplicates, a header that is also a shared header, and the header the auth mode already writes. Shared headers, per-user headers, secret URL variables and the auth mode combine on one registry. Omitted on update keeps the declarations; [] removes them.

Runtime — the credential resolver reads the caller's value per declared header (keyed by principal.Subject and the registry instance, so PRE and PRO hold separate values and one registry bound to several applications reuses the value).

  • Missing required value → -32003 with the connect link (cause: user_header_missing); tools/list shows the server as needs_connect.
  • The application itself and group keys are refused (application_not_connected) before the vault is read.
  • Upstream sessions, the discovery cache and introspection are scoped per principal; the pin key carries a digest of the values so a replaced key dials a new session.
  • Upstream 401 on a registry whose auth mode is none: the value that was sent is marked refused (user_header_rejected) and the person is asked to enter it again; no retry. With any other auth mode the 401 is returned as is, since it cannot tell which credential failed.
  • Inbound client headers never reach the upstream, so a client cannot override the value.

Enrollment

  • Self-service: the connect page lists the servers that need the person's keys with a form to save or remove them. Secrets are never written back into the page; an empty secret field keeps the stored value. Writes are accepted only from the gateway's own page.
  • Admin-issued link: POST /v1/gateways/{gw}/registries/{id}/user-headers/links (consumer_path required for a custom server).
  • Bulk import: PUT …/user-headers/values (≤1000 rows, every row validated before any write).

Administration — GET …/user-headers lists holders (written / refused timestamps, never values); DELETE …/user-headers/values[?principal_sub=] resets one person or everyone. Deleting a registry removes its values.

Storage — values live only in the shared Redis vault (the store the DB-less MCP data plane reads; the control plane reaches it through the existing fallback), so a change made on the data plane never leaves a stale copy elsewhere. The vault gains list/delete by provider prefix (Postgres, Redis, fallback). No migration: declarations ride the existing mcp_target JSON and config-sync snapshot.

Audit — mcp_user_header_{set,removed,rejected,reset,imported,link_issued} with gateway, registry, opaque principal reference, header name and actor; never the value.

Commits are one work unit each. docs(api) regenerates OpenAPI and also picks up A2A locator fields already on develop that had not been regenerated.

Rollout

Additive. A data plane older than this build ignores user_headers and dials without them (the upstream answers 401; no other person's key is ever sent). Deploy TrustGate (CP + DP) before the console starts writing user_headers. No feature flag.

Values live in Redis: deployments must run it with persistence (or an external Redis), otherwise a Redis restart drops every enrolled key.

Not in this PR

Console UI, product docs, last-use tracking, a retention/purge policy, 401 handling when the auth mode also sends a credential. The app-key + X-NeuralTrust-End-User path inherits ENG-1790.

Test plan

  • go test ./...
  • golangci-lint and gosec -severity high on every touched package (0 issues)
  • go vet -tags functional ./tests/...
  • Functional E2E locally, header-authenticated upstream registered as a custom server:
    • app key + end user: application itself refused; missing key → -32003 with link; key entered on the connect page; key loaded by an admin; each call carries only the caller's own key; a client-sent X-API-Key never reaches the upstream; roster never shows a key; 401 → re-entry without retry; admin reset
    • OAuth inbound: person enrolls through an admin-issued link with consumer_path
    • DB-less MCP data plane sends a key an admin loaded on the control plane, and the reset reaches it
    • DEBUG logs grepped: no key appears; audit events present
  • Full local functional suite: only TestA2A_RefusesMethodsItDoesNotForward, TestA2A_PolicyBlocksTheMessage and TestA2A_AuthenticatesAndRoutes fail, and they fail identically on a clean develop checkout
  • CI
  • DEV QA with a real header-authenticated MCP server

CI note

develop currently fails lint on pkg/app/a2a/translated.go (staticcheck QF1002), which this PR does not touch, so the lint check is expected to fail for that reason. Commits were made with --no-verify for the same reason, after running lint, gosec and the test suite by hand.

An MCP target can now declare upstream headers whose value belongs to each
caller (user_headers), next to the shared headers map. A declaration names
the header, whether it is required or secret, and an optional scheme.

Validation rejects reserved and gateway-owned names, duplicates, a header
that is also shared, and the header the auth mode already writes. Values
are bounded and refuse CR, LF, NUL and non-ASCII. Values are keyed in the
vault by registry instance and header, so PRE and PRO keep separate values.

Refs ENG-3243
The credential resolver now reads the calling principal's value for every
per-user header a registry declares and sets it after the shared headers
and the auth mode. A required header without a value refuses the call with
a connect link (-32003); an optional one is left out. The application
itself and group keys are refused before the vault is read.

Upstream sessions, the discovery cache and introspection are scoped per
principal, and the pin key carries a digest of the values so a replaced
key dials a new session. An upstream 401 on a registry whose only
per-caller credential is its headers marks the values refused and asks the
caller to enter them again, without retrying.

Refs ENG-3243
mcp_target.user_headers is read on create and update and returned on read.
An update that omits it keeps the stored declarations and an empty list
removes them. A catalog row with per-user headers counts as customized, so
a catalog reapply never rewrites it and drops them.

Refs ENG-3243
The connect page lists every server of the consumer whose upstream needs
the person's own header values, with a form to enter, replace or remove
them. A secret value is never written back into the page, and leaving its
field empty keeps what is stored. Every value is validated before any is
written. An application's ticket shows no such server, and a ticket pinned
to one instance reaches only that instance.

Writes are accepted only from the gateway's own page, like the OAuth start,
and each change is audited with the header name and an opaque principal
reference, never the value.

Refs ENG-3243
Admins can now list who holds a value for an instance's per-user headers
(when it was written and whether the upstream refused it, never the value),
reset one person's values or everyone's, load values in bulk (up to 1000
rows, all validated before any is written) and issue one person the
connect link that enters their own values.

Values an admin writes on the control plane go to both vault stores, since
the MCP data plane reads only the shared Redis. Deleting a registry removes
every value entered for it. The vault gains listing and deletion by provider
prefix across principals for Postgres, Redis and the fallback composition.

The self-service connect page writes through the same path and audit.

Refs ENG-3243
Adds mcp_target.user_headers and the registry user-headers routes. The
regeneration also picks up the A2A agent locator fields already on develop
that were not regenerated there.

Refs ENG-3243
A header-authenticated MCP upstream registered as a custom server, reached
by one application on behalf of two end users. Covers: the application
itself refused, a missing key answered with a connect link, a key entered on
the connect page and one loaded by an admin, each call carrying only the
caller's own key, a client-sent header never reaching the upstream, the
admin roster never showing a key, a 401 marking the key for re-entry
without retrying it, and an admin reset.

Refs ENG-3243
A 401 does not say which credential failed. Per-user header values are now
marked refused only on a registry whose auth mode sends nothing of its own,
and only when the stored value is the one that was sent, so a key replaced
during the call is kept and an admin's static key or a caller's token never
sends everyone back to re-enter theirs.

A caller with no individual identity now reads that the server takes each
person's own key, instead of the shared-account remedy that does not apply.

Refs ENG-3243
The connect page held up to five seconds for a focused server with no
account row, taking it for one not yet synced. A server whose only
credential is its per-user keys is here; its key rows count as found.

Refs ENG-3243
Values were written to Postgres and Redis, so a key a person removed or
replaced on the MCP data plane (which reaches only Redis) left a stale copy
the control plane kept reading. They are now written only to the shared
Redis behind the control-plane vault, which both planes read, as the data
plane's OAuth credentials already are.

An admin link for a custom server now requires consumer_path: the Store
serves only the servers it materialised, so a Store link showed nothing.

Refs ENG-3243
…nbound

Adds a DB-less MCP plane receiving a key an admin loaded on the control
plane, and a person signed in with the IdP who enrolls through an
admin-issued link for a custom server.

Refs ENG-3243
@edukmx edukmx self-assigned this Nov 13, 2026
Keeps both the per-user header and the secret store admin routes, and
regenerates the OpenAPI docs over the merged tree. Renames the functional
test upstream helper that develop also introduced.

Refs ENG-3243
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Trivy code scan — HIGH/CRITICAL

No HIGH or CRITICAL vulnerabilities found. (run)

The per-user header value fixture looked like a live API key. Renamed, and
the old commit's fingerprint is added to .gitleaksignore since the scan
reads every commit.

Refs ENG-3243
…calls

Registering an agent now probes its endpoint with a tasks/get (GetTask) for
a task that cannot exist. The fake agent recorded that probe as a caller's
call, so the tests asserting a refused or blocked call never reached the
agent failed. It now answers the probe as an agent would, without recording
it.

Refs ENG-3243
Takes develop's .gitleaksignore, which already ignores the same fixture
fingerprint, so the file is identical on both branches.

Refs ENG-3243

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant